Medical diffusion on a budget

Textual Inversion for medical image generation

Bram de Wilde, Anindo Saha, Maarten de Rooij, Henkjan Huisman, Geert Litjens

🚧 TODO

add image(s) to title slide
debug r-stack, I think I need to export to png
potentially a slide on that FID/MFID is a bad metric

Diffusion models

Popular for text-to-image modeling

Stable Diffusion: open source + inference on single GPU

🚨 Training requires a lot of compute and data

🏥 Medical domain can have rare diseases and local compute

Prostate MRI?

🧐 Does Stable Diffusion already know medical imaging?

✒️ “a prostate MRI scan”

✒️ “a T2-weigthed MRI scan of a prostate”

💡 Fine-tune model

Fine-tuning diffusion models

Methods fine-tune sub-parts of diffusion model

✒️ Textual Inversion only trains token embedding

Textual Inversion example

Adapt to medical imaging

Original work uses small embeddings and ~5 examples

💡 Use larger embeddings and more examples

Classification

🚧 image visualizing setup

Classification

#Real	#Synthetic	AUC - Prostate MRI
200	0	0.780 ± 0.017
200	2000	0.803 ± 0.009

📈 Adding synthetic cases maintains or improves performance

Classification

#Real	#Synthetic	AUC - Prostate MRI
200	0	0.780 ± 0.017
200	2000	0.803 ± 0.009
0	2000	0.766 ± 0.020

📈 Adding synthetic cases maintains or improves performance

📉 Using only synthetic data gives small performance drop

Classification

#Real	#Synthetic	AUC - Prostate MRI
200	0	0.780 ± 0.017
200	2000	0.803 ± 0.009
0	2000	0.766 ± 0.020
0	2000*	0.562 ± 0.036

📈 Adding synthetic cases maintains or improves performance

📉 Using only synthetic data gives small performance drop

👎 Training on 10-case embeddings shows quality difference

Comparison to GAN baseline

🧐 What about GANs?

🔬 Fine-tune a pre-trained StyleGAN3 on 100 images

⚖️ Similar training time and compute

🧑‍⚕️ Prostate radiologist preferred diffusion model (36/50)

Composing embeddings

↔︎️ Interpolate between healthy and diseased state

⚡ Train two embeddings: healthy, diseased

✒️ “healthy:30% AND diseased:70%”

Composing embeddings

➕ Combine multiple embeddings to show multiple diseases

⚡ Train embeddings per disease

✒️ “pleural_effusion AND pneumonia”

Disease inpainting

Curate synthesized dataset
Precise control over disease location

IMAGE (full prostate image in a couple fragments)

Future directions

Fine-tune medical diffusion model
Applicable to 3D text-to-image models
Study utility for rare diseases
Exploit controlled synthesis with composing and inpainting

Thanks for listening!

🚧 style this page more

Scan for paper, code, contact details & more!

Chambon et al.

Bonus slide showing what happens if you have data + compute